Adjusting motion capture based on the distance between tracked objects

ABSTRACT

The technology disclosed relates to adjusting the monitored field of view of a camera and/or a view of a virtual scene from a point of view of a virtual camera based on the distance between tracked objects. For example, if the user&#39;s hand is being tracked for gestures, the closer the hand gets to another object, the tighter the frame can become—i.e., the more the camera can zoom in so that the hand and the other object occupy most of the frame. The camera can also be reoriented so that the hand and the other object remain in the center of the field of view. The distance between two objects in a camera&#39;s field of view can be determined and a parameter of a motion-capture system adjusted based thereon. In particular, the pan and/or zoom levels of the camera may be adjusted in accordance with the distance.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/187,186, entitled “Adjusting Motion Capture Based On the DistanceBetween Tracked Objects”, filed 21 Feb. 2014 (Attorney Docket No. LEAP1034-2/LPM-015US), which claims the benefit of U.S. provisional PatentApplication No. 61/767,997, entitled, “ADJUSTING MOTION CAPTURE BASED ONTHE DISTANCE BETWEEN TRACKED OBJECTS,” filed on Feb. 22, 2013 (AttorneyDocket No. LEAP 1034-1/LPM-015PR). The provisional application is herebyincorporated by reference for all purposes.

FIELD OF THE TECHNOLOGY DISCLOSED

Implementations of the technology disclosed generally relate tocomputer-based motion-tracking systems and, more particularly, toadjusting a tracking camera in accordance with tracked objects.

BACKGROUND

Traditionally, users have interacted with electronic devices (such as acomputer or a television) or computing applications (such as computergames, multimedia applications, or office applications) via indirectinput devices, including, for example, keyboards, joysticks, or remotecontrollers. The user manipulates the input devices to perform aparticular operation, such as selecting a specific entry from a menu ofoperations. Modern input devices, however, include multiple buttons,often in a complex configuration, to facilitate communication of usercommands to the electronic devices or computing applications; correctoperation of these input devices is often challenging to the user.Additionally, actions performed on an input device generally do notcorrespond in any intuitive sense to the resulting changes on, forexample, a screen display controlled by the device. Input devices canalso be lost, and the frequent experience of searching for misplaceddevices has become a frustrating staple of modern life.

An alternative mode of interaction involves recognizing and tracking theintentional movement of a user's hand, body, or any other object as itperforms a gesture, which can be interpreted by the electronic device asuser input or a command. For example, a motion-capture system can trackthe position of an object by acquiring one or more images of a spatialregion that includes the object, panning or zooming the image-capturedevice so that the object remains in the field of view.

Many sophisticated or nuanced gestures or motions, however, cannoteasily be tracked, identified, or interpreted by these systems. A usercan make large, broad gestures one moment followed by small, fine-tuninggestures. The capturing camera and/or supporting system may not be ableto react or reconfigure itself quickly enough to capture, or assignmeaning to, both kinds of gestures in quick succession. If the camera iszoomed out, for example, it can miss the subtleties of small gestures,whereas if the camera is zoomed in, it can fail to capture largermotions that stray outside the field of view. A need therefore existsfor systems and methods capable of responsively adjusting to gesturesthat rapidly change in scale.

SUMMARY

The technology disclosed relates to adjusting the monitored field ofview of a camera and/or a view of a virtual scene from a point of viewof a virtual camera based on the distance between tracked objects. Forexample, if the user's hand is being tracked for gestures, the closerthe hand gets to another object, the tighter the frame can become i.e.,the more the camera can zoom in so that the hand and the other objectoccupy most of the frame. The camera can also be reoriented so that thehand and the other object remain in the center of the field of view. Thedistance between two objects in a camera's field of view can bedetermined and a parameter of a motion-capture system adjusted basedthereon. In particular, the pan and/or zoom levels of the camera may beadjusted in accordance with the distance; for example, the camera canzoom in to view a small distance or zoom out to view a large distance. Acomputer display can be similarly adjusted—that is, a representation ofthe objects (and/or graphical features controlled by the objects) on thedisplay can be zoomed in for small distances and out for largedistances.

These and other objects, along with advantages and features of thetechnology disclosed herein disclosed, will become more apparent throughreference to the following description, the accompanying drawings, andthe claims. Furthermore, it is to be understood that the features of thevarious implementations described herein are not mutually exclusive andcan exist in various combinations and permutations.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1A schematically illustrates a motion-capture system in accordancewith an implementation of the technology disclosed.

FIG. 1B illustrates a block diagram of a computer system implementing animage-analysis system in accordance with an implementation of thetechnology disclosed.

FIGS. 2A-2C illustrate objects and distances therebetween for use inconnection with implementations of the technology disclosed.

FIG. 3 shows a method of controlling a camera using a distance of twoobjects in space.

FIG. 4 depicts a representative method of controlling zoom level of acamera responsive to distance between two objects in a 3D monitoredspace.

FIG. 5 illustrates one implementation of a method of controlling zoomlevel of a virtual camera responsive to distance between two objects ina 3D monitored space.

FIG. 6 is a flowchart showing a method of adapting a display betweencoarse and fine control movements responsive to distance between twocontrol objects in a 3D monitored space.

FIG. 7 illustrates a flowchart showing a method of adjusting a field ofview for capturing movement of objects within a monitored space.

DESCRIPTION

As used herein, a given signal, event or value is “responsive to” apredecessor signal, event or value of the predecessor signal, event orvalue influenced by the given signal, event or value. If there is anintervening processing element, step or time period, the given signal,event or value can still be “responsive to” the predecessor signal,event or value. If the intervening processing element or step combinesmore than one signal, event or value, the signal output of theprocessing element or step is considered “dependent on” each of thesignal, event or value inputs. If the given signal, event or value isthe same as the predecessor signal, event or value, this is merely adegenerate case in which the given signal, event or value is stillconsidered to be “dependent on” the predecessor signal, event or value.“Dependency” of a given signal, event or value upon another signal,event or value is defined similarly.

Referring first to FIG. 1A, which illustrates an exemplarygesture-recognition system 100A including any number of cameras 102, 104coupled to an image-analysis system 106. Cameras 102, 104 can be anytype of camera, including cameras sensitive across the visible spectrumor, more typically, with enhanced sensitivity to a confined wavelengthband (e.g., the infrared (IR) or ultraviolet bands); more generally, theterm “camera” herein refers to any device (or combination of devices)capable of capturing an image of an object and representing that imagein the form of digital data. While illustrated using an example of a twocamera implementation, other implementations are readily achievableusing different numbers of cameras or non-camera light sensitive imagesensors or combinations thereof. For example, line sensors or linecameras rather than conventional devices that capture a two-dimensional(2D) image can be employed. The term “light” is used generally toconnote any electromagnetic radiation, which may or may not be withinthe visible spectrum, and can be broadband (e.g., white light) ornarrowband (e.g., a single wavelength or narrow band of wavelengths).

Cameras 102, 104 are preferably capable of capturing video images (i.e.,successive image frames at a constant rate of at least 15 frames persecond); although no particular frame rate is required. The capabilitiesof cameras 102, 104 are not critical to the technology disclosed, andthe cameras can vary as to frame rate, image resolution (e.g., pixelsper image), color or intensity resolution (e.g., number of bits ofintensity data per pixel), focal length of lenses, depth of field, etc.In general, for a particular application, any cameras capable offocusing on objects within a spatial volume of interest can be used. Forinstance, to capture motion of the hand of an otherwise stationaryperson, the volume of interest can be defined as a cube approximatelyone meter on a side.

In some implementations, the illustrated system 100A includes one ormore sources 108, 110, which can be disposed to either side of cameras102, 104, and are controlled by image-analysis system 106. In oneimplementation, the sources 108, 110 are light sources. For example, thelight sources can be infrared light sources, e.g., infraredlight-emitting diodes (LEDs), and cameras 102, 104 can be sensitive toinfrared light. Use of infrared light can allow the gesture-recognitionsystem 100A to operate under a broad range of lighting conditions andcan avoid various inconveniences or distractions that can be associatedwith directing visible light into the region where the person is moving.However, a particular wavelength or region of the electromagneticspectrum can be required. In one implementation, filters 120, 122 areplaced in front of cameras 102, 104 to filter out visible light so thatonly infrared light is registered in the images captured by cameras 102,104. In another implementation, the sources 108, 110 are sonic sourcesproviding sonic energy appropriate to one or more sonic sensors (notshown in FIG. 1A for clarity sake) used in conjunction with, or insteadof, cameras 102, 104. The sonic sources transmit sound waves to theuser; the user either blocks (or “sonic shadowing”) or alters the soundwaves (or “sonic deflections”) that impinge upon her. Such sonic shadowsand/or deflections can also be used to detect the user's gestures and/orprovide presence information and/or distance information using rangingtechniques known in the art. In some implementations, the sound wavesare, for example, ultrasound, that is not audible to humans (e.g.,ultrasound).

It should be stressed that the arrangement shown in FIG. 1A isrepresentative and not limiting. For example, lasers or other lightsources can be used instead of LEDs. In implementations that includelaser(s), additional optics (e.g., a lens or diffuser) can be employedto widen the laser beam (and make its field of view similar to that ofthe cameras). Useful arrangements can also include short- and wide-angleilluminators for different ranges. Light sources are typically diffuserather than specular point sources; for example, packaged LEDs withlight-spreading encapsulation are suitable.

In operation, light sources 108, 110 are arranged to illuminate a regionof interest 112 that includes a control object portion 114 (in thisexample, a hand) that can optionally hold a tool or other object ofinterest and cameras 102, 104 are oriented toward the region 112 tocapture video images of the hand 114. In some implementations, theoperation of light sources 108, 110 and cameras 102, 104 is controlledby the image-analysis system 106, which can be, e.g., a computer system,control logic implemented in hardware and/or software or combinationsthereof. Based on the captured images, image-analysis system 106determines the position and/or motion of object 114.

FIG. 1B is a simplified block diagram of a computer system 100B,implementing image-analysis system 106 (also referred to as an imageanalyzer) according to an implementation of the technology disclosed.Image-analysis system 106 can include or consist of any device or devicecomponent that is capable of capturing and processing image data. Insome implementations, computer system 100B includes a processor 132,memory 134, a sensor interface 136, a display 138 (or other presentationmechanism(s), e.g. holographic projection systems, wearable googles orother head mounted displays (HMOs), heads up displays (HUDs), othervisual presentation mechanisms or combinations thereof, speakers 139, akeyboard 140, and a mouse 141. Memory 134 can be used to storeinstructions to be executed by processor 132 as well as input and/oroutput data associated with execution of the instructions. Inparticular, memory 134 contains instructions, conceptually illustratedas a group of modules described in greater detail below, that controlthe operation of processor 132 and its interaction with the otherhardware components. An operating system directs the execution oflow-level, basic system functions such as memory allocation, filemanagement and operation of mass storage devices. The operating systemcan be or include a variety of operating systems such as MicrosoftWINDOWS operating system, the Unix operating system, the Linux operatingsystem, the Xenix operating system, the IBM AIX operating system, theHewlett Packard UX operating system, the Novell NETWARE operatingsystem, the Sun Microsystems SOLARIS operating system, the OS/2operating system, the BeOS operating system, the MAC OS operatingsystem, the APACHE operating system, an OPENACTION or OPENSTEP operatingsystem, iOS, Android or other mobile operating systems, or anotheroperating system platform.

The computing environment can also include otherremovable/non-removable, volatile/nonvolatile computer storage media.For example, a hard disk drive can read or write to non-removable,nonvolatile magnetic media. A magnetic disk drive can read from or writeto a removable, nonvolatile magnetic disk, and an optical disk drive canread from or write to a removable, nonvolatile optical disk such as aCD-ROM or other optical media. Other removable/non-removable,volatile/nonvolatile computer storage media that can be used in theexemplary operating environment include, but are not limited to,magnetic tape cassettes, flash memory cards, digital versatile disks,digital video tape, solid state RAM, solid state ROM, and the like. Thestorage media are typically connected to the system bus through aremovable or non-removable memory interface.

Processor 132 can be a general-purpose microprocessor, but depending onimplementation can alternatively be a microcontroller, peripheralintegrated circuit element, a CSIC (customer-specific integratedcircuit), an ASIC (application-specific integrated circuit), a logiccircuit, a digital signal processor, a programmable logic device such asan FPGA (field-programmable gate array), a PLD (programmable logicdevice), a PLA (programmable logic array), an RFID processor, smartchip, or any other device or arrangement of devices that is capable ofimplementing the actions of the processes of the technology disclosed.

Camera interface 136 can include hardware and/or software that enablescommunication between computer system 100B and cameras such as cameras102, 104 shown in FIG. 1A, as well as associated light sources such aslight sources 108, 110 of FIG. 1A. Thus, for example, camera interface136 can include one or more data ports 146, 148 to which cameras can beconnected, as well as hardware and/or software signal processors tomodify data signals received from the cameras (e.g., to reduce noise orreformat data) prior to providing the signals as inputs to amotion-capture (“mocap”) program 144 executing on processor 132. In someimplementations, camera interface 136 can also transmit signals to thecameras, e.g., to activate or deactivate the cameras, to control camerasettings (frame rate, image quality, sensitivity, etc.), or the like.Such signals can be transmitted, e.g., in response to control signalsfrom processor 132, which can in turn be generated in response to userinput or other detected events.

Camera interface 136 can also include controllers 147, 149, to whichlight sources (e.g., light sources 108, 110) can be connected. In someimplementations, controllers 147, 149 provide operating current to thelight sources, e.g., in response to instructions from processor 132executing mocap program 144. In other implementations, the light sourcescan draw operating current from an external power supply, andcontrollers 147, 149 can generate control signals for the light sources,e.g., instructing the light sources to be turned on or off or changingthe brightness. In some implementations, a single controller can be usedto control multiple light sources.

Instructions defining mocap program 144 are stored in memory 134, andthese instructions, when executed, perform motion-capture analysis onimages supplied from cameras connected to sensor interface 136. In oneimplementation, mocap program 144 includes various modules, such as anobject detection module 152, an object analysis module 154, and adistance-analysis module 156. Object detection module 152 can analyzeimages (e.g., images captured via camera interface 136) to detect edgesof an object therein and/or other information about the object'slocation. Object analysis module 154 can analyze the object informationprovided by object detection module 152 to determine the 3D positionand/or motion of the object (e.g., a user's hand). Distance analysismodule 156 can analyze, in the manner set forth below, two or moreobjects detected by module 152 to determine the distance between them.Examples of operations that can be implemented in code modules of mocapprogram 144 are described below.

Display 138, speakers 139, keyboard 140, and mouse 141 can be used tofacilitate user interaction with computer system 100B. In someimplementations, results of gesture capture using camera interface 136and mocap program 144 can be interpreted as user input. For example, auser can perform hand gestures that are analyzed using mocap program144, and the results of this analysis can be interpreted as aninstruction to some other program executing on processor 132 (e.g., aweb browser, word processor, or other application). Thus, by way ofillustration, a user might use upward or downward swiping gestures to“scroll” a webpage currently displayed on display 138, to use rotatinggestures to increase or decrease the volume of audio output fromspeakers 139, and so on.

It will be appreciated that computer system 100B is illustrative andthat variations and modifications are possible. Computer systems can beimplemented in a variety of form factors, including server systems,desktop systems, laptop systems, tablets, smart phones or personaldigital assistants, wearable devices, e.g., goggles, head mounteddisplays (HMDs), wrist computers, and so on. A particular implementationcan include other functionality not described herein, e.g., wired and/orwireless network interfaces, media playing and/or recording capability,etc. In some implementations, one or more cameras can be built into thecomputer or other device into which the sensor is imbedded rather thanbeing supplied as separate components. Further, an image analyzer can beimplemented using only a subset of computer system components (e.g., asa processor executing program code, an ASIC, or a fixed-function digitalsignal processor, with suitable I/O interfaces to receive image data andoutput analysis results).

While computer system 100B is described herein with reference toparticular blocks, it is to be understood that the blocks are definedfor convenience of description and are not intended to imply aparticular physical arrangement of component parts. Further, the blocksneed not correspond to physically distinct components. To the extentthat physically distinct components are used, connections betweencomponents (e.g., for data communication) can be wired and/or wirelessas desired.

Referring now also to FIGS. 2A-2C, the object detection module 152 ofthe motion-capture system 100 identifies at least two objects 202, 204.One object 202 can be a hand of the user; the second object 204 can beanother hand (as illustrated), a fixed object (e.g., a button, table, orcontrol console), or a moving object (e.g., a wand or stylus). Thetechnology disclosed is not, however, limited to any particular type ofobject. The two objects 202, 204 can be identified by a user (by, e.g.,inputting a command to inform the system 100 which objects to track, bytouching the object (which is sensed and interpreted by mocap 144program as a designation), or otherwise signaling selection of one ormore objects) or can be dynamically identified by the system 100 itself(e.g., by determining that the distance between the two objects 202, 204is changing, by identifying the objects 202, 204 as the dominant objectsin the region 112). The distance analysis module 156 dynamicallymeasures the distance 206 between the two objects 202, 204 (the“object-to-object distance”).

Measuring the object-to-object distance typically includes calculating,inferring, or otherwise determining the real-world spatial coordinatesof each of the objects 202, 204. The cameras 102, 104 operate inconjunction with one or both of the light sources 108, 110 to provide 2Dimages of a viewed scene. For example, one light source 108 and twocameras 102, 104 can be used, or two light sources 108, 110 and onecamera 102. From these 2D images, the system 100 determines the 3Dposition of the objects 202, 204. In particular, the distance analysismodule 156 can derive the object-to-object distance 206 from the imagesthemselves, from the 3D data generated by the system 100, or from otherdata generated therefrom (e.g., from 3D models constructed from the 3Ddata).

In various implementations, the system 100 changes one or more of itsbehaviors or parameters based on the object-to-object distance. Forexample, the measured distance 206 can be used to zoom and/or pan one orboth of the cameras 102, 104. As used herein, the term “zoom” meansaltering the focal length (and thus the angle of view) of a camera'soptics, and the term “pan” means shifting the field of view of thecamera, e.g., by rotating the camera around an axis. Panning and/orzooming the cameras 102, 104 can improve the accuracy of the system 100by, for example, zooming in to enlarge a portion of a scene to betterview details if the object-to-object distance is small, or zooming outof a scene to retain two designated objects within the field of view ifthe object-to-object distance is large. The zoom levels of the cameras102, 104 can be adjusted optically, digitally, or by any other means ofincreasing or decreasing the zoom level of the acquired images; forexample, in response to commands issued by distance analysis module 156,the cameras 102, 104 can reduce or increase the focal length of theirlenses or can digitally magnify or de-magnify the images.

Alternatively or in addition, the measured distance 206 can be used tochange what is shown on the display 138. The display 138 can display aview of a virtual scene from the point of view of a virtual camera, andthe distance 206 can be used to change the position of the virtualcamera relative to the virtual scene. The display 138 can also or inaddition display virtual elements, symbols, or other graphics that canchange (in size, shape, view, or sensitivity with respect to a usercommand) based on the distance 138. For example, if the measureddistance 206 is small, the display 138 can be altered to show a close-upview of the objects 202, 204 (and/or interface elements related to theobjects), and if the measured distance 206 is large, the display can bealtered to show a wider-angle view of the objects 202, 204 (and/orinterface elements related to the objects); in other words, zoomingand/or panning can be achieved by adjustment of the cameras, or byadjustment of the display contents, or by adjusting the cameras and thedisplay contents. The close-up view of the objects 202, 204 on thedisplay 138 can permit a user to make smaller, more precise movements orgestures involving the objects 202, 204, while the wide-angle view canpermit a user to make broader, coarser movements or gestures withoutleaving the field of view. In various implementations, the display 138displays direct representations of the objects 202, 204 (e.g., a set ofvirtual, on-screen “hands” that mimic the size, shape, position,orientation, and movement of a user's real hands) or symbolicrepresentations of the objects 202, 204 (e.g., a set of on-screen styli,points, or other symbols) that represent the position, orientation,and/or movement of a user's real hands). In other implementations, thedisplay 138 does not display a direct or symbolic representation of theobjects 202, 204; instead, the displayed contents react to the motion ofthe objects 202, 204. For example, the display 138 can present a globethat rotates in response to motion of the objects 202, 204 (e.g., handgestures that suggest imparting spin), but does not include direct orsymbolic representations of the objects 202, 204 themselves. In all ofthese cases, what is presented on the display 138 is considered “relatedto” the objects 202, 204 and/or motion thereof.

The system 100 can alternatively or in addition interpret a gesture bythe user involving one or both of the objects 202, 204 differently basedon the distance 206; for example, the sensitivity of the system 100 togesture distances can be increased at small distances 206 to allow auser to have more precise control. A rotational movement of the twoobjects 202, 204, for example, can have more of an effect (i.e., cause agreater on-screen movement) if the two objects 202, 204 are far apart(e.g., 0.5 to one meter apart) and less of an effect (i.e., cause asmaller on-screen movement) if the two objects 202, 204 are closetogether (e.g., one to ten centimeters apart). For example, if the twoobjects 202, 204 are far apart, their rotation though a 90° movement cancause an on-screen map to rotate through 90°; if, however, the twoobjects 202, 204 are close together, their rotation through 90° cancause the on-screen map to rotate an amount less than 90° (e.g.,) 45°.

In one implementation, the distance analysis module 156 measures theobject-to-object distance 206 as the shortest distance between the twoobjects 202, 204 (i.e., the distance d between the points R₁ on thefirst object 202 and R₂ on the second object 204 that are closest toeach other), as shown in FIG. 2C (200C). In another implementation, theobject-to-object distance 206 can be measured as the distance betweenselected reference points 208, 210 on each object 202, 204, as shown inFIG. 2A (200B); these reference points 208, 210 can be determined by theuser or by the object analysis module 154. For example, the user candesire that a point on the index finger is always tracked as thereference point of a hand. The user can even set reference points onadditional fingers of the hand, thereby indicating that each fingershould be tracked as a separate object and the object-to-objectdistances measured accordingly; in such cases, the object-to-objectdistance used in determining a pan and/or zoom adjustment can be theaverage of these individual distances. In one implementation, as shownin FIG. 2B (200B), the object analysis module 154 determines thereference point of a recognized object as the average position of thecenter of the object. For example, the center of a user's palm can bethe reference point on the user's hand.

The distance analysis module 156 can be configured to continuouslyadjust the zoom level (and/or other parameters of the system 100) as theobject-to-object distance 206 changes. Alternatively, the parameters canbe adjusted only after the distance 206 crosses a threshold. Forexample, if one of the objects being tracked is a hand, a slightmovement, such as a hand tremor, can trigger an undesired zoomadjustment. The threshold value can be set by the user or by thedistance analysis module 156 (e.g., using a filter or based on thefrequency of detected movements, with higher-frequency movements assumedto be spurious); in one implementation, the threshold must be crossedbefore the parameters are adjusted. After the threshold value iscrossed, the parameter can be adjusted continuously or in accordancewith an updated threshold. In one implementation, after the threshold isreached, the parameter is continuously updated until the rate of changeof object-to-object distance 206 is at zero, or near zero, for aprerequisite amount of time, at which time the distance is comparedagain to the same or an updated threshold. A hysteretic delay betweenadjustments can be imposed to avoid excessive, visually distractingtransitions, particularly if the frequency at which the threshold iscrossed is high.

In another implementation, the parameter of the system 100 can beadjusted based on the rate of change of the object-to-object distance206 in addition to, or instead of, the absolute object-to-objectdistance 206. If, for example, the rate of change is high, the view canbe zoomed in or out quickly; if the rate of change is low, the distanceanalysis module 156 may not trigger a zoom level adjustment until thechange in object-to-object distance 206 and/or its rate of changecrosses a predetermined threshold value. The paths of one or both of theobjects 202, 204 can be predicted and, if the prediction indicates achange in the distance 206, the parameter of the system 100 can bechanged accordingly.

In one implementation, the distance analysis module 156 analyzes the 3Dpositions of objects 202, 204 as determined by object analysis module154 to measure the distance of the objects 202, 204 to the cameras 102,104 (the “object-to-camera distance”). The object analysis module 154can thus be configured to trigger zoom level adjustment (or theadjustment of other parameters) based on the object-to-camera distancein addition to (or instead of) the object-to-object distance. Forexample, a decreasing object-to-object distance (e.g., the distance 206)can result in a greater change in zoom level at a large object-to-cameradistance than if the same object-to-object distance were observed at asmall object-to-camera distance. In other words, an object-to-objectdistance reduction of one centimeter might be clearly observable if theobjects are only one meter from the cameras, but difficult to observe attwenty meters from the cameras, and the zoom level is adjustedaccordingly.

In addition to tracking the distance 206 between the objects 202, 204,the distance analysis module 156 can also track the positions of theobjects 202, 204. If the objects approach an edge of the field of viewof the cameras 102, 104, for example, or an object represented on thedisplay 134 approaches an edge thereof, the mocap program 144 can signalcameras 102, 104 to pan in order to track the object. As noted above,panning can refer to separately or congruently moving the individualfields of view of the cameras 102, 104 in any direction. The distanceanalysis module 156 can trigger panning based on the location on theobjects 202, 204 to keep objects 202, 204 within the field of view.Alternatively, the distance analysis module 156 can calculate anddynamically track a center point (centroid) 212 (referring again to FIG.2A) between the objects 202, 204; panning can then be based on thelocation of the center point 212. For example, objects 202, 204 can moveto the right at the same rate—thereby shifting the location of thecenter point 212 to the right—without altering the distance betweenthem. This motion results in the cameras 102, 104 panning to right, butthe zoom level remains constant. Alternatively, just one object 202 canmove to the right while object 204 stays stationary. Accordingly, thesystem pans to the right and can also zoom out as the center point 212has shifted to the right and the object-to-object distance 206 hasincreased. The motion-capture device can be configured to continuouslytrigger panning to keep the center point 212 at the center of thedefined boundary, or it can be configured to only trigger panning oncethe center point 212 moves beyond a smaller boundary set within thisboundary.

The motion-capture system 100 can be operable to run in a plurality ofzoom level and panning control modes as described above to effectivelytrack multiple objects of interest, and to adjust this tracking based onthe level of information, feedback, or sensitivity required by the user,application of use, or system capabilities. Additionally, the distancesmeasured by the distance analysis module 156 as described can be actualphysical distances or the same concepts can be implemented using virtualdistances, such as measured pixels.

Flowcharts

FIG. 3 shows a method 300 of controlling a camera using a distance oftwo objects in space. Flowchart 300 can be implemented at leastpartially with and/or by one or more processors configured to receive orretrieve information, process the information, store results, andtransmit the results. Other implementations may perform the actions indifferent orders and/or with different, fewer or additional actions thanthose illustrated in FIG. 3. Multiple actions can be combined in someimplementations. For convenience, this flowchart is described withreference to the system that carries out a method. The system is notnecessarily part of the method.

At action 302, first and second objects are identified in space. In oneimplementation, the first and second objects are at least partiallywithin a field of view of the camera. In other implementations, thefirst and second objects are entirely within a field of view of thecamera.

At action 312, positional information of the first and second objects isdetermined from a distance between the first and second objects inspace. In one implementation, the distance corresponds to an averagedistance between (i) a point on the first object closest to the secondobject and a point on the second object closest to the first object,(ii) a selected point on the first object and a selected point on thesecond object or (iii) a centroid of the first object and a centroid ofthe second.

At action 322, one or more camera parameters are adjusted based at leastin part upon the distance determined. In one implementation, an updateddistance for the objects is repeatedly determined from new positionalinformation for the first and second objects and the one or more cameraparameters are adjusted based at least in part upon the updated distancedetermined. In another implementation, a position of the field of viewof the camera is adjusted based at least in part upon the updateddistance determined. In some implementations, a zoom of the camera isadjusted based at least in part upon the updated distance determined. Inanother implementation, a focal length of the camera is adjusted basedat least in part upon the updated distance determined.

FIG. 4 depicts a representative method 400 of controlling zoom level ofa camera responsive to distance between two objects in a 3D monitoredspace. Flowchart 400 can be implemented at least partially with and/orby one or more processors configured to receive or retrieve information,process the information, store results, and transmit the results. Otherimplementations may perform the actions in different orders and/or withdifferent, fewer or additional actions than those illustrated in FIG. 4.Multiple actions can be combined in some implementations. Forconvenience, this flowchart is described with reference to the systemthat carries out a method. The system is not necessarily part of themethod.

At action 402, first and second objects in a field of view of the cameraare identified. In some implementations, first and second objects areidentified as objects to be tracked responsive to selection of the firstand second objects by a touch gesture. In other implementations, firstand second objects are identified as objects to be tracked responsive tochanging distance between first and second objects.

At action 412, spatial coordinates of the objects in the 3D monitoredspace are repeatedly calculated and also the distance between theobjects. In some implementations, the first object is a hand of a userand the second object is another hand of the user. In otherimplementations, the first object is a hand of a user and the secondobject is a fixed object. In yet other implementations, the first objectis a hand of a user and the second object is a moving object. In someother implementations, a specification can be received from a user forreference points on the objects dependent on which distance between theobjects is calculated.

At action 422, a focal length of the camera is altered responsive toresponsive to the calculated distance. In some implementations, thefocal length is decreased when the distance between the tracked objectsincreases or crosses a predetermined threshold distance. In otherimplementations, the focal length is increases when the distance betweenthe tracked objects decreases or crosses a predetermined thresholddistance. Yet other implementations include altering a focal length ofthe camera responsive to rate of change of the calculated distance. Someother implementations include calculating distance between the objectsand the camera and responsive to the calculated distance between theobjects and the camera and/or between the objects, altering a focallength of the camera.

At action 432, the field of view of the camera is moved responsive tothe calculated distance by rotating the camera around an axis, asdescribed above in the application.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. In the interest ofconciseness, the combinations of features disclosed in this applicationare not individually enumerated and are not repeated with each base setof features. The reader will understand how features identified in thissection can readily be combined with sets of base features identified asimplementations.

Other implementations can include a non-transitory computer readablestorage medium storing instructions executable by a processor to performany of the methods described above. Yet another implementation caninclude a system including memory and one or more processors operable toexecute instructions, stored in the memory, to perform any of themethods described above.

FIG. 5 illustrates one implementation of a method 500 of controllingzoom level of a virtual camera responsive to distance between twoobjects in a 3D monitored space. Flowchart 500 can be implemented atleast partially with and/or by one or more processors configured toreceive or retrieve information, process the information, store results,and transmit the results. Other implementations may perform the actionsin different orders and/or with different, fewer or additional actionsthan those illustrated in FIG. 5. Multiple actions can be combined insome implementations. For convenience, this flowchart is described withreference to the system that carries out a method. The system is notnecessarily part of the method.

At action 502, first and second objects in a field of view of a realworld camera are identified. In some implementations, first and secondobjects are identified as objects to be tracked responsive to selectionof the first and second objects by a touch gesture. In otherimplementations, first and second objects are identified as objects tobe tracked responsive to changing distance between first and secondobjects.

At action 512, spatial coordinates of the objects in the 3D monitoredspace are repeatedly calculated and also the distance between theobjects. In some implementations, the first object is a hand of a userand the second object is another hand of the user. In otherimplementations, the first object is a hand of a user and the secondobject is a fixed object. In yet other implementations, the first objectis a hand of a user and the second object is a moving object. In someother implementations, a specification can be received from a user forreference points on the objects dependent on which distance between theobjects is calculated.

At action 522, a view of a virtual scene from a point of view of thevirtual camera is altered responsive to responsive to the calculateddistance. In some implementations, the view of the virtual scene ismoved responsive to the calculated distance. In other implementations,the view of the virtual scene is narrowed responsive to reduction thecalculated distance. In yet other implementations, the view of thevirtual scene is widened responsive to increase the calculated distance.

At action 532, the virtual elements of the virtual scene are alteredresponsive to the calculated distance. In one implementation, size ofvirtual elements in the virtual scene is adjusted responsive to thecalculated distance. Some implementations include adjusting shape ofvirtual elements in the virtual scene responsive to the calculateddistance. Other implementations include adjusting responsiveness ofvirtual elements in the virtual scene responsive to the calculateddistance.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 6 is a flowchart showing a method 600 of adapting a display betweencoarse and fine control movements responsive to distance between twocontrol objects in a 3D monitored space. Flowchart 600 can beimplemented at least partially with and/or by one or more processorsconfigured to receive or retrieve information, process the information,store results, and transmit the results. Other implementations mayperform the actions in different orders and/or with different, fewer oradditional actions than those illustrated in FIG. 6. Multiple actionscan be combined in some implementations. For convenience, this flowchartis described with reference to the system that carries out a method. Thesystem is not necessarily part of the method.

At action 602, first and second objects in a field of view of a realworld camera are identified. In some implementations, first and secondobjects are identified as objects to be tracked responsive to selectionof the first and second objects by a touch gesture. In otherimplementations, first and second objects are identified as objects tobe tracked responsive to changing distance between first and secondobjects.

At action 612, spatial coordinates of the objects in the 3D monitoredspace are repeatedly calculated and also the distance between theobjects. In some implementations, the first object is a hand of a userand the second object is another hand of the user. In otherimplementations, the first object is a hand of a user and the secondobject is a fixed object. In yet other implementations, the first objectis a hand of a user and the second object is a moving object. In someother implementations, a specification can be received from a user forreference points on the objects dependent on which distance between theobjects is calculated.

At action 622, responsiveness of one or more virtual elements togestures by the first or second control object is altered such that thegestures more precisely control the virtual elements. Someimplementations include altering responsiveness of one or more virtualelements to gestures by the first or second control object such that thegestures by the first or second control object more coarsely control thevirtual elements responsive to increase in the calculated distance.Other implementations include increasing responsiveness of the virtualelements responsive to increase in the calculated distance. Yet otherimplementations include decreasing responsiveness of the virtualelements responsive to reduction in the calculated distance.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 7 illustrates a flowchart showing a method 700 of adjusting a fieldof view for capturing movement of objects within a monitored space.Flowchart 700 can be implemented at least partially with and/or by oneor more processors configured to receive or retrieve information,process the information, store results, and transmit the results. Otherimplementations may perform the actions in different orders and/or withdifferent, fewer or additional actions than those illustrated in FIG. 7.Multiple actions can be combined in some implementations. Forconvenience, this flowchart is described with reference to the systemthat carries out a method. The system is not necessarily part of themethod.

At action 702, digital images including first and second objects in themonitored space are received from at least one camera, as describedabove in this application. At action 712, information related to thefirst or second objects is displayed on an electronic display, asdescribed above in this application.

At action 722, a distance between the first and second objects isdetermined using information derived from the digital images, asdescribed above in this application. In one implementation, the distancecorresponds to an average distance between (i) a point on the firstobject closest to the second object and a point on the second objectclosest to the first object, (ii) a selected point on the first objectand a selected point on the second object or (iii) a centroid of thefirst object and a centroid of the second.

At action 732, a parameter of at least one of (i) the at least onecamera or (ii) the electronic display is modified in accordance with thedetermined distance. In some implementations, the modified parameter isa zoom level of the at least one camera or an orientation of the atleast one camera relative to the monitored space. One implementationincludes the adjusted parameter being a view displayed on the electronicdisplay. In one implementation, the first object is a hand of a user andthe second object is an object identified by the user. In anotherimplementation, the parameter is adjusted only when the distance crossesa predetermined threshold distance. In yet another implementation,threshold distance is based at least in part on a distance between theat least one camera and the first or second object. In someimplementations, the zoom level is adjusted by zooming in or out at arate based at least in part on a rate of change of the distance. Inother implementations, modification of the parameter corresponds topanning the camera based on a position of the first object or the secondobject. In one implementation, the camera is panned so as to track (i) amidpoint of the distance between the first and second objects, (ii) thefirst object, or (iii) the second object.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

The terms and expressions employed herein are used as terms andexpressions of description and not of limitation, and there is nointention, in the use of such terms and expressions, of excluding anyequivalents of the features shown and described or portions thereof. Inaddition, having described certain implementations of the technologydisclosed, it will be apparent to those of ordinary skill in the artthat other implementations incorporating the concepts disclosed hereincan be used without departing from the spirit and scope of thetechnology disclosed. Accordingly, the described implementations are tobe considered in all respects as only illustrative and not restrictive.

What is claimed is:
 1. A method of controlling a camera using a distanceof two objects in a 3D monitored space, the method including:identifying from images captured by the camera first and second objectspresent in the 3D monitored space, the first and second objects at leastpartially within a field of view of the camera; repeatedly calculatingspatial coordinates of the first and second objects present in the 3Dmonitored space based upon the images captured by the camera andcalculating a distance between the first and second objects in spacefrom the spatial coordinates; and responsive to the calculated distance,adjusting a camera parameter; whereby the camera is enabled to captureadditional images of the first and second objects moving in spacerelative to the camera.
 2. The method of claim 1, further including:establishing a threshold value to be set for the calculated distance;detecting the threshold value is crossed by the calculated distance; andresponsive to detecting the threshold value is crossed, performing theadjusting on the camera parameter continuously, after the threshold isreached.
 3. The method of claim 1, further including: using a filter toeliminate spurious changes in the calculated distance detected in theimages.
 4. The method of claim 1, wherein the adjusting includes:adjusting a zoom of the camera based at least in part upon thecalculated distance.
 5. The method of claim 1, wherein the adjustingincludes: adjusting a focal length of the camera based at least in partupon the calculated distance.
 6. The method of claim 1, wherein theadjusting includes: moving the field of view of the camera responsive tothe calculated distance.
 7. The method of claim 1, further includingidentifying first and second objects as objects to be tracked responsiveto selection of the first and second objects by a touch gesture.
 8. Themethod of claim 1, further including identifying first and secondobjects as objects to be tracked responsive to changing distance betweenfirst and second objects.
 9. The method of claim 1, wherein the firstobject is a first finger of a hand of a user and the second object isanother finger of the hand of the user.
 10. The method of claim 1,wherein the first object is a hand of a user and the second object isanother hand of the user.
 11. The method of claim 1, wherein the firstobject is a hand of a user and the second object is a fixed object. 12.The method of claim 1, wherein the first object is a hand of a user andthe second object is a moving object.
 13. The method of claim 1, furtherincluding receiving a specification from a user for reference points onthe objects dependent on which distance between the objects iscalculated.
 14. The method of claim 1, further including: determining arate of change for the calculated distance; and altering the cameraparameter responsive to the rate of change of the calculated distance.15. The method of claim 1, further including: calculating distancebetween the objects and the camera; and responsive to the calculateddistance between the objects and the camera and/or between the objects,altering the camera parameter.
 16. A non-transitory computer readablestorage medium impressed with computer program instructions to control avirtual camera responsive to distance between two objects in a 3Dmonitored space, the instructions, when executed on a processor,implement a method including: identifying first and second objectsphysically present in a field of view of a real world camera from imagescaptured by the real world camera of the 3D monitored space; repeatedlycalculating spatial coordinates of the first and second objectsphysically present in the 3D monitored space based upon the imagescaptured by the camera and calculating a distance between the first andsecond objects from the spatial coordinates; and responsive to thecalculated distance between the first and second objects physicallypresent in a field of view of a real world camera, altering a view of avirtual scene from a point of view of the virtual camera.
 17. Thenon-transitory computer readable storage medium of claim 16,implementing the method further including moving the view of the virtualscene responsive to the calculated distance.
 18. The non-transitorycomputer readable storage medium of claim 16, implementing the methodfurther including narrowing the view of the virtual scene responsive toreduction in the calculated distance.
 19. The non-transitory computerreadable storage medium of claim 16, implementing the method furtherincluding widening the view of the virtual scene responsive to increasein the calculated distance.
 20. The non-transitory computer readablestorage medium of claim 16, implementing the method further includingadjusting size of virtual elements in the virtual scene responsive tothe calculated distance.
 21. The non-transitory computer readablestorage medium of claim 16, implementing the method further includingadjusting shape of virtual elements in the virtual scene responsive tothe calculated distance.
 22. The non-transitory computer readablestorage medium of claim 16, implementing the method further includingadjusting responsiveness of virtual elements in the virtual sceneresponsive to the calculated distance.
 23. A non-transitory computerreadable storage medium impressed with computer program instructions toadapt a display between coarse and fine control movements responsive todistance between two control objects in a 3D monitored space, theinstructions, when executed on a processor, implement a methodincluding: identifying first and second control objects in a field ofview of a camera; repeatedly calculating spatial coordinates of thecontrol objects in the 3D monitored space and distance between thecontrol objects; and responsive to reduction in the calculated distance,altering responsiveness of one or more virtual elements to gestures bythe first or second control object such that the gestures more preciselycontrol the virtual elements.
 24. The non-transitory computer readablestorage medium of claim 23, implementing the method further including,responsive to increase in the calculated distance, alteringresponsiveness of one or more virtual elements to gestures by the firstor second control object such that the gestures by the first or secondcontrol object more coarsely control the virtual elements.
 25. Thenon-transitory computer readable storage medium of claim 23,implementing the method further including increasing responsiveness ofthe virtual elements responsive to increase in the calculated distance.26. The non-transitory computer readable storage medium of claim 23,implementing the method further including decreasing responsiveness ofthe virtual elements responsive to reduction in the calculated distance.